Add an encoding parameter to io.load_tabby
#116
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
By default, `Path.open()` uses `locale.getencoding()` when opening the file for reading. This has caused problems when loading files saved (presumably on Windows) with iso-8859-1 encoding on linux (where utf-8 is the default), see psychoinformatics-de#112 The default behaviour is maintained with `encoding=None`, and any valid encoding name can be provided as an argument to load_tabby. The encoding will be used for loading tsv files. The encoding is stored as an attribute of `_TabbyLoader` rather than passed as an input to the load functions - since they may end up being called in a few places (when sheet import statements are found), it would be too much passing around otherwise. With external libraries it might be possible to guess a file encoding that produces a correct result based on the files content, but the success is not guaranteed when there are few non-ascii characters in the entire file (think: list of authors). Here, we do not attempt to guess, instead expecting the user to know the encoding they need to use. Ref: https://docs.python.org/3/library/pathlib.html#pathlib.Path.open https://docs.python.org/3/library/functions.html#open
mslw
added a commit
to sfb1451/tabby-utils
that referenced
this pull request
Nov 21, 2023
This relies on a not-yet-merged patch to datalad-tabby, so we call load_tabby without additional arguments unless the encoding is given. If the patch gets merged, the if-else can be removed. Related datalad-tabby change: psychoinformatics-de/datalad-tabby#116
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR resolves #112 by adding an optional
encodingparameter toio.load_tabby. The parameter can be used to specify encoding for reading tsv files.When not specified (
encoding=None), we keep the default behavior (implicitly usinglocale.getencoding()1,2).With external libraries it might be possible to guess a file encoding that produces a correct result based on the files content, but the success is not guaranteed when there are few non-ascii characters in the entire file (think: list of authors). I made an attempt with #114 but didn't like it in the end. Here, we do not attempt to guess, instead expecting the user to know the encoding they need to use.
This PR also fixes an unrelated documentation typo to satisfy the codespell checks.
Footnotes
https://docs.python.org/3/library/pathlib.html#pathlib.Path.open ↩
https://docs.python.org/3/library/functions.html#open ↩